Image generating apparatus, image generating method, and recording medium

ABSTRACT

An image generating apparatus which generates a second image that is an image converted from a first image captured by an imaging apparatus placed to be oriented downward at an angle of depression with respect to a horizontal direction includes: a specifying unit which specifies a straight line included in the first image and passing through an intersection between an imaging plane and a vertical vector; and an image extension processing unit which generates the second image by sampling pixels of the first image along the specified straight line.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is based on and claims priority of JapanesePatent Application No. 2010-279946 filed on Dec. 15, 2010. The entiredisclosure of the above-identified application, including thespecification, drawings and claims is incorporated herein by referencein its entirety.

BACKGROUND OF THE INVENTION

(1) Field of the Invention

The present invention relates to perspective distortion correctionperformed on an image captured by a monitoring apparatus and so on.

(2) Description of the Related Art

Conventionally, solid-state imaging elements such as a charge coupleddevice (CCD) and a complementary metal oxide semiconductor (CMOS) havebeen used for monitoring apparatuses. Widely used as such monitoringapparatuses is a monitoring apparatus which performs image processing ona captured image so as to detect an object, particularly, a person. As amethod of detecting a person in the image, a method using a patternrecognition technique based on statistical learning as disclosed inNon-Patent Reference 1 is generally used (Dalai, N. and Triggs, B.,“Histograms of oriented gradients for human detection”, Computer Visionand Pattern Recognition, 2005. IEEE Computer Society Conference, pages:886-893).

In this method, each of the following techniques is used. A featureselection technique is used for extracting, from the captured image, afeature quantity used for identification. An identifier constructiontechnique is used for constructing an identifier which takes theselected feature quantity as input and determines whether or not theselected feature quantity represents the target. A technique todetermine, using the constructed identifier, whether or not there is aperson in a partial area in the imaging area is used.

A monitoring apparatus using a normal lens is provided at a constantangle of depression in a high position in a ceiling and so on. Thus, inthe image captured from above, a significant perspective distortion iscaused as a result of projecting a three-dimensional subject onto atwo-dimensional imaging element. This causes a difference in tilt of aperson in an image portion, even if it is the image portion of the sameperson standing upright, depending on whether the image portion of theperson is located in a center of the image or at an end portion. Inother words, this causes a difference in direction of the person (adirection from a foot to a head, and so on).

However, the identifier constructed for human detection is only capableof detecting a person having a particular tilt in the image. Therefore,to automatically detect all people in the image captured by themonitoring apparatus, a method generally used is to generate identifierscorresponding to tilts of all the people that can be captured in theimage and perform judgment processing the number of times equivalent tothe number of identifiers.

In addition, in the monitoring apparatus, to respond to the needs formonitoring an entire range with one camera, an omnidirectional imagingapparatus using a fisheye lens, an omnidirectional mirror, and so on hascome to be used more often than before. In the case of the monitoringapparatus using the omnidirectional imaging apparatus, a specialcircular image is captured, so that the detection is often performed onan image generated by converting the captured image.

For example, in Non-Patent Reference 2 (Tatsuo Sato, Kazuhiro Goto,“Extension Method for Omni-directional Camera”, OITA-AIST Joint ResearchCenter, Industrial Technology Div., Vol: 2002, pages 9 to 11), panoramicextension is performed on an omnidirectional image which maintains anaspect ratio of an object, based on a center of the circular image. As aresult, this equalizes the directions of the upper and lower parts ofthe person in the extended image, thus allowing the detection to beperformed using the same processing as the processing for an imagecaptured with a normal lens.

SUMMARY OF THE INVENTION

However, in the case of using more identifiers, there is a problem thatsuch human detection processing requires more storage area andprocessing time.

In addition, even in the monitoring apparatus using a normal lens, awide-angle lens (with a view angle of approximately 90 degrees) is oftenused so as to secure a monitoring range as wide as possible. However,since perspective distortion is more significant in the monitoringapparatus using the wide-angle lens than in the monitoring apparatususing the normal lens, it is necessary to provide more detectors when itis intended to detect all the people in the captured image.

Furthermore, the image conversion performed on the omnidirectional imagehas a problem that such image conversion cannot be used for the imagingapparatus using the normal lens because Non-Patent Reference 2 is basedon the premise that the input image is an omnidirectional image.

The present invention is to solve the conventional problem describedabove, and it is an object of the present invention to provide animaging apparatus which performs image conversion processing appropriatefor object detection by equalizing tilts of the people in the image.

To solve the conventional problem as described above, an imagegenerating apparatus according to an aspect of the present invention isan image generating apparatus which generates a second image that is animage converted from a first image captured by an imaging apparatusplaced to be oriented downward at an angle of depression with respect toa horizontal direction, and the image generating apparatus includes: aspecifying unit which specifies a straight line included in the firstimage and passing through an intersection between an imaging plane and avertical vector having a start point at the imaging apparatus; and animage extension processing unit which generates the second image bysampling pixels of the first image along the specified straight line.

With this configuration, it is possible to generate an image in whichtilts of people (objects) captured in the image, which derive from theperspective distortion and the method of capturing the omnidirectionalimage, are integrated through image conversion.

With the imaging apparatus according to an implementation of the presentinvention, since the tilts of the people (objects) standing upright withrespect to the ground are unified in the output image, object detection,particularly, human detection requires only one feature quantity.

In addition, the optical system can output an appropriate correctedimage even when the image is captured using, not limited to a wide-anglelens, a fisheye lens or an omnidirectional mirror.

It is possible to specify an appropriate direction (see a direction 4bd) by preventing an inappropriate direction (see a direction 9 xd) frombeing specified despite an optical axis of the imaging apparatus (seedirection 804L in FIG. 8) being directed diagonally downward (seedescription in FIG. 8 and so on).

This further allows simplifying the configuration by preventingcomplexity of configuration (see human detector 100 d), despite theoptical axis being directed diagonally downward.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other objects, advantages and features of the invention willbecome apparent from the following description thereof taken inconjunction with the accompanying drawings that illustrate a specificembodiment of the present invention. In the Drawings:

FIG. 1 is a block diagram showing a configuration of an imagingapparatus according to a first embodiment;

FIG. 2 is a diagram showing an example of the imaging apparatusaccording to the first embodiment;

FIG. 3 is a diagram showing an example of the imaging apparatusaccording to the first embodiment;

FIG. 4 is a diagram showing a projection onto an imaging plane through alens;

FIG. 5 is a diagram showing a state of projection using a pinholecamera;

FIG. 6 is a diagram showing the imaging plane placed in front of avirtual viewpoint position.

FIG. 7 is a diagram showing an example of a captured image and anextended image that is output by an image conversion processing unitaccording to the first embodiment;

FIG. 8 is a diagram in which an under-camera point, equal depressionangles, and half lines connecting them are drawn on the captured imageaccording to the first embodiment;

FIG. 9 is a diagram in which the half lines are drawn on the capturedimage according to the first embodiment;

FIG. 10 is a diagram showing an image conversion method according to thefirst embodiment;

FIG. 11 is an extended image with a constant sampling interval whenperforming image conversion according to the first embodiment;

FIG. 12 is an extended image with a sampling interval to give a locallyequal aspect ratio when performing image conversion according to thefirst embodiment;

FIG. 13 is a conceptual diagram showing that tilts of objects areunified when performing image extension according to the firstembodiment;

FIG. 14 is a diagram showing an operation flow of the imaging apparatusaccording to the first embodiment;

FIG. 15 is a block diagram showing a configuration of an imagingapparatus according to a second embodiment;

FIG. 16 is a diagram showing a projection method of a fisheye lens;

FIG. 17 is a diagram showing an example of the imaging apparatusaccording to the second embodiment;

FIG. 18 is a diagram showing a projection method for a normal lens(central projection model); and

FIG. 19 is a diagram showing an example of an image generated bycorrecting a captured image by a distortion correcting unit, accordingto the second embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, embodiments of the present invention will be described withreference to the drawings. Note that each of the embodiments describedbelow shows a specific example of a preferred implementation of thepresent invention. Any constituent element, connection mode of theconstituent elements, steps, and order of steps are mere examples, andthese are not intended to limit the present invention. The presentinvention is limited only by What is claimed is. Accordingly, of theconstituent elements in the embodiments below, some constituent elementsthat are not described in independent claims representing highestconcepts of the present invention are described as not essential but ascomponents of a more preferable embodiment.

An image generating apparatus according to the embodiment is an imagegenerating apparatus (image conversion apparatus, computer, imageprocessing apparatus, image conversion processing unit) 100 whichgenerates a second image (extended image) 3 which is generated byperforming image conversion on a first image (captured image) 2 that iscaptured by an imaging apparatus 101 placed to be oriented downward atan angle of depression (angle 201) with respect to a horizontaldirection (direction x in FIG. 2), and the image generating apparatus100 includes: a specifying unit 105 x which specifies a line 803 (FIG.8) included in the first image 2 (FIG. 2, upper column in FIG. 7, rightcolumn in FIG. 8) and passing through an intersection (under-camerapoint) 801 between an imaging plane 807 and a vector (unit under-cameravector) 806 in a vertical direction (direction z in FIG. 8) having anendpoint at the imaging apparatus 101; and an image extension processingunit 103 which generates the second image 3 by sampling pixels from thefirst image 2 along the specified line 803.

More specifically, for example, by specifying a plurality of pixelsincluded in the line 803, the line 803 made up of these pixels isspecified.

For example, pixels included, in the second image 3, in a line 5 bL in atop-bottom direction 3 y of the second image 3 as a result of thesampling described above are pixels that have been sampled to beincluded in the specified line 803. More specifically, as a result ofsampling the pixels from the line 803L that are included in the firstimage 2, the direction 4 bd of a person 100 h in a first image portion 4bI that is a first image portion in the same direction as a direction803L of the line 803 is converted to an appropriate direction (direction2 y) as described above. This converts a direction 5 bd of the person100 h in a second image portion 5 bI to an appropriate direction afterchange (direction 2 y), thus simplifying the configuration of thedetector.

In addition, for example, the first image 2 to be captured includes theimage portion 4 bI which includes image of a person 4 b (in FIG. 7), thedirection 4 bd from a foot to a head of the person 4 b included in theimage portion is different from the top-bottom direction 2 y of thefirst image 2; this different direction 4 bd is the same direction asthe direction of the line 803 passing through an intersection 801 of avertical vector 806 and an imaging plane 807 (the first image 2projected onto the imaging plane 807) and through a position of theimage portion 4 bI; and the specification unit 105 x specifies thedirection 803L of the specified line 803 as the direction 4 bd of theperson 4 b; the second image 3 to be generated includes an image portion5 bI including image of a person 5 b that is the same person as theperson 4 b; and the included image portion 5 bI is an image portiongenerated by converting the specified direction 803L (direction 4 bd) inthe image portion 4 bI included in the first images 2 to the top-bottomdirection 3 y of the second image 3.

With this, despite being captured downward at the angle (angle ofdepression, installation angle of depression) 201, the image portion 4bI is converted to the image portion 5 bI, so as to prevent detection ofthe person 4 b located in the inappropriate direction 4 bd in the image4 bI and detect the person 5 b in the appropriate direction 5 bd in theimage portion 5 bI. This allows simplifying the configuration used forthe detection (for example, human detector 100 d in FIG. 2).

Note that the inappropriate direction is, for example, a directiondifferent from the direction (direction 4 ad) of another person (person4 a) that is other than one person (person 4 b) to be detected.

In other words, the appropriate direction is a direction that is thesame direction as the direction (direction 5 ad) of the other person(person 5 a) that is not the one person (person 5 b), and so on.

Note that the direction (directions 4 ad and 5 ad) of the other personin the image (the first image 2, the second image 3) is, for example,the top-bottom direction (directions 2 y and 3 y) of the image.

Note that the imaging plane 807, for example, is an area onto which thefirst image 2 is projected. This imaging plane 807 may be, for example,a plane on which the captured first image 2 is projected when the firstimage 2 is captured by an imaging element provided in the imaging plane807. In addition, the imaging plane 807, for example, is not a plane inwhich the imaging element is provided, but may simply be a plane inwhich in the case of providing an imaging element and so on, the firstimage 2 is assumed to be projected onto the imaging element.

Note that the imaging plane 807 is, for example, a plane in which anoptical axis (see a vector 805 in FIG. 8) of the imaging apparatus 101passes through a center (see position 9 x in FIG. 3) of the plane, andwhich is perpendicular to the optical axis.

Thus, a first plane, which is the imaging plane 807 when the angle 201at which the imaging apparatus 101 is placed to be oriented downward isan angle 201 b that is another angle different from one angle 201 a(FIG. 2), is different from a second plane that is the imaging plane 807when the angle 201 is the one angle 201 a.

In other words, one line (see one direction 803La) that is the line 803specified when the angle 201 is the other angle 201 b is different fromthe other line that is the line 803 specified when the angle 201 is oneangle 201 a.

The data representing the angle 201 may be provided (see the parameterholding unit 104 in FIG. 1 and so on).

For example, the data may be held by the image generating apparatus 100or may be obtained by an obtaining unit 100 g (FIG. 2) from the imagingapparatus 101.

Note that the angle 201 indicated by the obtained data (see data 101 din FIG. 2) may be, for example, an angle detected by a detecting unit(gyroscope and so on) 101 s included in the imaging apparatus 101.

The data may be, for example, the data that is input to the imagegenerating apparatus 100 by a user of the system 100 s and so onoperating on the image generating apparatus 100.

When the obtained data indicates the one angle 201 a, it is possible tospecify, from the pixels in the first line, the pixels included in theline 5 bL in the top-bottom direction 3 y in the generated second image3, as being the pixels included in one line specified in this case andincluded in the first image 2, and so on.

When the obtained data indicates the other angle 201 b, it is possibleto specify, from the pixels in the other line, the pixels included inthe line 5 bL in the top-bottom direction 3 y in the generated secondimage 3, as being the pixels included in the other line specified in thecase and included in the first image 2, and so on.

With this, in both cases where the angle 201 is the one angle 201 a andwhere the angle 201 is the other angle 201 b, it is possible to generatean appropriate image for the second image 3.

Note that, for example, the angle 201 may be the one angle 201 a whenthe imaging apparatus 101 is installed in one place, and may be theother angle 201 b when the imaging apparatus 101 is installed in theother place.

In addition, for example, the angle 201 may be the one angle 201 aduring one period, and may be the angle 201 b during another period thatis a subsequent period following the one period.

Note that accordingly a center position 9 x of the first image 2 and adirection 9 xd of a line (straight line) passing through the position ofa first image portion 4 bI are assumed.

This direction 9 xd is another direction that is not a directionrepresenting an upward direction of the vertical direction z (a negativedirection of the direction z) in a space 100 r in which a person 100 his present (see the “base point” and the “center” in Non-PatentReference 2 described earlier).

In this context, the direction 9 xd is specified, and processing basedon the specified direction 9 xd is performed, so that it is possible toavoid inappropriate processing to be performed by the processing basedon the other direction.

In other words, this specifies the direction 803L of the line 803passing through the point 801 in the vertical direction z from theimaging apparatus 101 and the position of the first image portion 4 bIon the first image 2 projected onto the imaging plane 807. This allowsperforming processing based on the specified direction 803L, thusallowing performance of appropriate processing.

First Embodiment)

With reference to FIGS. 1 to 19, the configuration of a system 100 saccording to a first embodiment is to be described.

FIG. 1 is a diagram showing the system 100 s in the first embodiment ofthe present invention.

The system 100 s includes an image conversion processing unit (imagegenerating apparatus) 100 and an imaging apparatus 101.

In FIG. 1, the image conversion processing unit 100 receives, as input,a captured image (first image) 2 captured by the imaging apparatus 101,and includes a captured image obtaining unit 102, an image extensionprocessing unit 103, a parameter holding unit 104, and a correspondingpoint calculating unit 105.

Note that part or all of the image conversion processing unit 100 maybe, for example, a computer including a CPU, a ROM, a RAM, and so on. Bycausing the computer to execute the program, each function to berealized by the image conversion processing unit 100, such as a functionof the captured image obtaining unit 102, may be realized in this imageconversion processing unit 100.

Note that the image conversion processing unit 100 may be part of theimaging apparatus 101.

In addition, the image conversion processing unit 100 may be provided,for example, outside of the imaging apparatus 101. For example, theimage conversion processing unit 100 may be provided in a space distantfrom a space to be imaged and monitored by the imaging apparatus 101,such as a building entrance. Such distant space may be, for example, aguard station of the building.

The imaging apparatus 101 is a monitoring apparatus which monitors, byitself, a relatively wide range. In this case, the imaging apparatus 101is a monitoring camera using a wide-angle lens (see a lens 101L in FIG.2 to be hereinafter described), which performs image capturing of themonitored space.

FIG. 2 is a diagram showing an example of the imaging apparatus 101.

FIG. 3 is a diagram showing an example of the captured image 2.

FIG. 3 shows an example of the captured image 2 when the imagingapparatus 101 is installed at a position higher than a floor surface 101f, at a certain installation angle of depression (angle) 201 as shown inFIG. 2. The captured image 2 is a digitized and recorded piece ofthree-dimensional spatial information (image) projected onto the imagingplane 807 of the imaging apparatus 101 (see FIGS. 4 to 6, FIG. 8, and soon to be hereinafter described).

Briefly described here are a projection of the three-dimensional spatialinformation onto the imaging plane 807 and a conversion between aphysical coordinate system on the imaging plane 807 and an imagecoordinate system on the captured image 2.

FIG. 4 is a diagram showing a state of projection performed by animaging apparatus using a model lens.

Light, which is emitted from a certain point in a three-dimensionalspace, is concentrated onto a certain point in an imaging plane gall,according to an incident angle onto the imaging plane 9 a 11. In themodel lens, a plurality of light collection points at which the light iscondensed and collected are concentrated on a plane (imaging plane 9 a11).

Thus, placing an imaging element such as a CCD on this plane allowscapturing a sharp image.

Such lens and imaging element causes a projection from thethree-dimensional space to the two dimensional space.

In this context, considering only the geometrical properties of theimaging apparatus, it is possible to approximate this imaging apparatusto a pinhole camera in a pinhole camera model by focusing only on asingle point in a center of the lens.

FIG. 5 is a diagram showing a state of projection by a pinhole camera.

The center of the lens is a virtual viewpoint position 9 a 3, and theplane to which the light is collected is an imaging plane 9 a 4. Inaddition, a distance between the virtual viewpoint position 9 a 3 andthe imaging plane 9 a 4 is a focal distance.

FIG. 6 is a diagram in which the imaging plane shown in FIG. 5 is placedin front of the virtual viewpoint position 9 a 3.

As shown in FIG. 5, in the pinhole camera, an object 9 a 2 is projectedupside down, but as shown in FIG. 6, when the imaging plane (imagingplane 9 a 5) is placed in front of the virtual viewpoint position 9 a 3,the object is projected without being reversed. Since similarity betweenobjects to be projected onto the imaging plane is ensured even if theimaging plane (imaging plane 9 a 5) is placed as shown in FIG. 6, theimaging plane is hereinafter considered to be located on the object 9 a2 side.

In an actual imaging apparatus, by providing, in the imaging plane, animaging element such as a CCD or CMOS, analog three-dimensional spaceinformation such as intensity of the light that has reached each elementin the imaging device is converted to a digital signal, to generate acaptured image.

Next, a coordinate (coordinate system) conversion between the physicalcoordinate system on the imaging plane and the image coordinate systemon the captured image is described.

It is possible to freely determine an origin, an aspect ratio, and so onof the image coordinate system on the captured image, independently ofthe physical coordinate system.

This requires, when modeling the actual camera, considering theconversion from the physical coordinate system to the image coordinatesystem.

For performing this conversion, it is only necessary to obtain atranslation for positioning the origin of the coordinate system and ascale conversion constant corresponding to an aspect ratio or a focallength.

Generally, these parameters are referred to as internal parameters, andare represented in form of a matrix.

Use of such matrix allows a conversion between image coordinates on thecaptured image and physical coordinates on the captured image. Withthis, considering a certain pixel in an image, it is possible tocalculate to find, in the physical coordinate system, through whichpoint on the imaging plane the light has passed to be collected to thevirtual viewpoint position.

In the image conversion processing unit 100, it is assumed that theinternal parameters in the imaging apparatus 101 are known.

At this time, the angle of depression (installation angle of depression,angle) 201 and the internal parameters are held by the parameter holdingunit 104. Note that, for example, the angle 201 may be one of theinternal parameters.

The imaging apparatus 101 transmits the captured image 2 generatedthrough the procedures above, to the image extension processing unit 103via the captured image obtaining unit 102 (FIG. 1).

The captured image obtaining unit 102, for example, is a recording andreproduction unit for the captured image 2.

The image extension processing unit 103 performs processing on thecaptured image 2 obtained from the captured image obtaining unit 102,and outputs an extended image (second image) 3 in which tilts of peoplein the captured image 2 are unified from the captured image (firstimage) 2.

FIG. 7 is an example of the extended image (second image) correspondingto the input captured image 2.

The image extension processing unit 103 generates an extended image 3 byperforming a certain image conversion (to be described in detail below),which is similar to processing of panoramic extension foromnidirectional images.

At this time, the corresponding point calculating unit 105 (FIG. 1)calculates each coordinate in the image (captured image 2, extendedimage 3) necessary for this conversion.

The content of the image extension processing will be described.

FIG. 8 is a diagram showing the captured image 2 and so on.

In FIG. 8, on the captured image 2, an under-camera point 801, an equaldepression angle point 802, and a half line 803 connecting these pointsare drawn.

FIG. 9 is a diagram in which a plurality of half lines 803 are drawn onthe captured image 2.

First, a method of calculating a corresponding point necessary for theimage conversion is described with reference to FIGS. 8 and 9.

The corresponding point calculating unit 105 calculates the under-camerapoint 801 and the equal depression angle point 802 in the captured image2. Subsequently, a half line 803 is created (specified) by setting anend point to the under-camera point 801 and extending a line from theend point to the equal depression angle point 802.

In FIG. 8, 804 represents a virtual viewpoint position (position of theimaging apparatus 101). 805 represents a unit optical-axis vector thatis a unit vector directed in the optical axis direction 804L of theimaging apparatus 101. 806 represents a unit under-camera vector that isa unit vector directed from a camera installation point toward avertically and directly downward direction (z direction).

In addition, 807 is assumed to represent an imaging plane of the imagingapparatus 101, on which the second captured image 2 is projected.

Note that the imaging plane 807 is of the same concept as the imagingplane in FIG. 6 and is different from the actual camera imaging plane;however, the image that can be obtained in this imaging plane 807 is animage whose similarity with the captured image 2 in the actual cameraimaging plane is ensured.

At this time, the under-camera point 801 is a point indicating anintersection (point 807 a in FIG. 8) between the unit under-cameravector 806 and the imaging plane 807, which is drawn on the capturedimage 2 by coordinate conversion using internal parameters.

Likewise, the equal depression angle point 802 is an intersection (point808 p) between the imaging plane 807 and a vector 808 (upper leftcolumn, in FIG. 8) rotated in a zenith angle direction according to asampling angle 901 that is a predetermined angle (FIG. 9), at an angleobtained by integral-multiplying this sampling angle 901, which is drawnon the captured image 2 by coordinate conversion.

Note that FIG. 9 illustrates, as an example, the vector 808 rotated atan angle obtained by tripling the sampling angle 901.

This sampling angle determines the intervals and the number of equaldepression angle points, and the number of half lines is determinedaccordingly.

Note that FIG. 9 illustrates 11 half lines among a plurality of the halflines to be specified.

As a result, this accordingly determines a horizontal resolution of theextended image 3.

The sampling angle 901 (FIG. 9) is held by the parameter holding unit104.

Note that in calculating the equal depression angle points 802, a unitoptical-axis vector 805 is used as an example, but it is only necessarythat each vector rotated in the zenith angle direction has the sameangle of depression, and it is not essential to use the unitoptical-axis vector 805. Note that as described above, for example, oneof the respective vectors has the same angle of depression as that ofanother vector, and the one vector is a vector generated by rotating theother vector, around the vertical direction z as a rotational axis.

FIG. 9 illustrates, on the captured image 2, an intersection (equaldepression angle point 802) between the imaging plane 807 and eachvector 808 obtained by rotating the unit optical-axis vector 805 in thezenith angle direction.

Note that FIG. 9 illustrates, on the captured image 2 as an example, anequal depression angle point 802 x obtained by rotating the unitoptical-axis vector 805 just by zero degree and directly extending theunit optical-axis vector 805. The specified half lines 803, for example,may include such half lines 803 at the equal depression angle point 802x that are obtained by directly extending the unit optical-axis vector805.

FIG. 10 is a diagram showing an image conversion method.

Next, a method of generating the extended image 3 using the half linesgenerated according to the under-camera point 801 that is the end pointof the half line 803 and the equal depression angle points 802 isdescribed with reference to FIG. 10.

First, when focusing on one half line 803, as shown in FIG. 10, assumingthat sampling is performed on the half line 803 on the captured image 2at certain intervals on the captured image 2 from an under-camera point801 side toward the equal depression angle point 802, the result of thesampling corresponds to one vertical line in the extended image 3 (seethe line 5 bL in FIG. 10). For the direction of the line 5 bL on theextended image 3, the under-camera point 801 side corresponds to a lowerside of the extended image 3 (a negative side of the direction 3 y), andan equal depression angle point 802 side corresponds to an upper side ofthe extended image 3 (a positive side of the direction 3 y).

In other words, the direction from the under-camera point 801 toward theequal depression angle point 802 in the captured image 2 is converted toa direction, in the extended image 3, from the lower side of theextended image 3 toward the upper side (positively directed toward thedirection (top-bottom direction) 3 y).

Note that for example, the line 5 bL in the top-bottom direction 3 y ofthe extended image 3 is considered.

The extended image 3 may include, in this line 5 bL, a pixel included inthe half line 803 in the captured image 2.

In other words, for example, the pixel in the line 5 bL as describedabove may have the same pixel value as that of the pixel in the halfline 803 or may be pixel a value of a neighboring pixel and the like.

The pixel value of the neighboring pixel is a pixel value or the likecorrected by approximating the pixel value of the pixel in the half line803 to a pixel value of the neighboring pixel thereof.

The extended image 3 may include, in this line 5 bL, each pixel in thehalf line 803 in the captured image 2.

The captured image 1 may include one pixel located at a closer side tothe under-camera point 801 (a negative side of the top-bottom direction2 y of the captured image 2) and another pixel on a farther side (a morepositive side in the top-bottom direction 2 y).

The extended image 3 may include the one pixel on a more negative sideof the top-bottom direction 3 y of the extended image 3 and the otherpixel on the more positive side.

Note that a technique of determining the sampling intervals on the halfline 803 (see intervals 1201 and 1202 in FIG. 12, and so on to behereinafter described) will be described later.

A plurality of sampling results on the half lines 803 are processed asshown by (1) to (11) in an upper portion (captured image 2) of FIG. 10and by (1) to (11) shown in a lower portion (captured image 3). In otherwords, the angles of the half lines 803 have an order based on theunder-camera point 801 of the captured image 2 as an origin. Each of thesampling results is included in the extended image 3 in the same orderas the order of the half lines 803 on which the sampling has beenperformed. In other words, the extended image 3 is generated byarranging the sampling results in these half lines 803 in the same orderas the order of the half lines 803, from left to right in a horizontaldirection (in the left-right direction 3 x).

In other words, each half line 803 in the captured image 2 has an order((1) to (11) in the upper part).

Each line 5 bL in the extended image 3 has an order ((1) to (11) in thelower part).

For example, pixels in an earlier half line 803 (sampling result) may beincluded in an earlier line 5 bL.

In addition, a horizontal resolution of the extended image 3 isdetermined by the number of half lines 803; FIG. 10 only illustratespart of the half lines 803, but in practice, half lines 803 equivalentin number to the horizontal resolution of the extended image 3 aregenerated.

FIG. 11 is a diagram showing an extended image 3 p that is an extendedimage 3 when setting constant sampling intervals (intervals 9 b) forimage conversion.

FIG. 12 is a diagram showing an extended image 3 q that is the extendedimage 3 when setting sampling intervals for image conversion to locallyequalize an aspect ratio.

Here, an example of determining the sampling intervals will be describedwith reference to FIGS. 11 and 12.

In the extended image 3, the resolution in a longitudinal direction (thedirection indicated by the top-bottom direction 3 y in FIG. 10) isdetermined by the sampling intervals in the half line 803 in thecaptured image 2.

In addition, the resolution in a cross direction is determined by thenumber of half lines 803, that is, the number of equal depression anglepoints 802.

Thus, setting the sampling intervals simply at a constant value as shownin FIG. 11 causes the aspect ratio of a person to change depending onwhich region among a plurality of regions of the extended image 3includes an image of the person (see the extended image 3 p in FIG. 11).

Note that for example, as shown in the extended image 3 p in FIG. 11, ina lower region in FIG. 11 which is a region on a relatively negativeside in the top-bottom direction 3 y, a ratio of a length in thelongitudinal direction (top-bottom direction 3 y) with respect to alength in the cross direction (left-right direction 3 x) is relativelyhigh.

On the other hand, this ratio is relatively low in an upper region whichis a region on a relatively positive side in the top-bottom direction 3y.

As a result, this requires more identifiers for human detection, so thatthe significance of performing image conversion is undermined.

Note that for example, this requires another identifier to perform thedetection in the upper region along with the identifier to perform thedetection in the lower region, thus causing increase in the number ofidentifiers.

Thus, as shown in FIG. 12, the number of half lines 803 and the samplingintervals on the half lines 803 are not determined independently of eachother, by equalizing a radius direction sampling interval 1201 in aradius direction (the direction of the half line 803) and acircumferential direction sampling interval 1202 in a circumferentialdirection (a circumferential direction centering on the point 801 thatis the end point of the half line 803) (for example, by setting a ratioto 1:1 as shown by two “Δr”s in FIG. 12). Accordingly, it is possible togenerate a more appropriate extended image 3 (extended image 3 q in FIG.12) in which an aspect ratio of the person is retained more sufficientlythan in the case of setting these numbers independently.

As a result, the extended image 3 appears as shown by the extended image3 q in FIG. 12, thus reducing the number of detectors for humandetection.

FIG. 13 is a conceptual diagram that indicates unification of tilts.

In addition, for the half line 803 connecting the under-camera point 801and each equal depression angle point 802, as shown in FIG. 13, theimage of the object or person vertical to the ground is includedconstantly along the half line 803 in the captured image 2. In otherwords, the direction of the person and so on (person 4 b and so on)captured in the extended image 3 (the direction 4 bd in FIG. 3) is thesame direction as the direction 803L (FIG. 13) of the half line 803passing through the position of the person and so on.

Thus, by sampling (extracting), along the half line 803, a plurality ofpixels included in the half line 803, it is possible to generate theextended image 3 in which the tilts of the people and so on (objects)are unified regardless of the positions of the people and so on in theimage.

Next, as described above, the operation of the image conversionprocessing unit 100 having the configuration as described above will bedescribed.

FIG. 14 is a flow diagram for describing the operation of the imageconversion processing unit 100.

Note that as will be described in detail later, for example, the angle201 (FIG. 2) may be changed from an angle before change (one angle 201a) to an angle after change (the other angle 201 b). In other words, asdescribed earlier, the imaging apparatus 101 is installed to be orienteddownward at the angle of depression 2 with respect to the horizontaldirection x (FIG. 2). For example, the optical axis of the installedimaging apparatus 101 is directed downward as described earlier. Thisangle 201 may be changed. For example, the processing in FIG. 14 to bedescribed may be performed based on the angle 201 after change (angle201 b), when the angle at which the apparatus is installed is changed tothe angle after change (angle 201 b).

In step ST1401, the corresponding point calculating unit 105 calculatesthe under-camera point 801 in the captured image 2 (described earlier),using the installation angle 201 (angle after change 201 a) and theinternal parameters.

In step ST1402, the corresponding point calculating unit 105 calculateseach equal depression angle point 802 in the captured image 2, using theinstallation angle 201, the internal parameters, and a sampling angle901 (FIG. 9).

In step ST1403, the image extension processing unit 103 generates theextended image 3, based on the captured image 2 received from thecaptured image obtaining unit 102, using the under-camera point 801 andthe equal depression angle points 802 (information 4 in FIG. 1) that arecalculated by the corresponding point calculating unit 105.

According to the first embodiment, regardless of the position in thecaptured image 2 in the monitoring apparatus, it is possible to generatean image (the second image 3 and so on in FIG. 10 and so on) having thesame tilt by unifying a tilt of each object or person at a positionthereof with a tilt of the person and so on at another position thereof.This only requires one feature quantity in object detection,particularly in human detection, thus facilitating the detectionprocessing.

Thus far, the system 100 s in the first embodiment of the presentinvention has been described, but the present invention is not limitedto this embodiment.

For example, in FIG. 1, the imaging apparatus 101 has been described asa monitoring camera using a wide-angle lens, but may also be amonitoring camera that captures all the directions using a hyperboloidalmirror, and it is only necessary to allow calculating the under-camerapoint and a group of equal depression angle points. In this case, theunder-camera point and the equal depression angle points in the capturedimage can be calculated by reflectance calculation of a light vector.

Second Embodiment

FIG. 15 is a block diagram showing a configuration of a system 1500 s.

With reference to FIG. 15, the configuration of the system 1500 s in thesecond embodiment is to be described.

Note that in FIG. 15, the same numerical references are used for thesame constituent elements as those in FIG. 1, and thus the detaileddescription thereof is omitted accordingly.

FIG. 15 shows an image conversion processing unit 1500 and a fisheyelens imaging apparatus 1501 according to the second embodiment.

In FIG. 15, 1502 is an image distortion correcting unit.

The fisheye lens imaging apparatus 1501 is an imaging apparatus whichallows imaging a wider range using a fisheye lens (see the lens 101L inFIG. 2) instead of a normal lens.

FIG. 16 is a diagram modeling a projection state of an incident light 9p onto an imaging plane 9 q, in an equidistant projection fisheye lens.

When f1 is a focal length of the fisheye lens, θ is an incident angle(zenith angle) of the light, and φ is a zenith angle, in the case ofequidistant projection, a distance L1 from the center of the imagingplane 9 q is determined to be proportional to the incident angle θ.

Accordingly, as shown in Expression (1), when a relationship between βand θ is determined, L1 and β has a relationship as represented byExpression (2).

In addition, L1 and imaging plane coordinates P1=(x1, y1) have arelationship as represented by Expressions (3) and (4).

Note that most fisheye lenses adopt the equidistant projection method.

[Math. 1]

θ=sin⁻¹ β  (1)

L₁=f₁β  (2)

x₁=L₁ cos φ  (3)

y₁=L₁ sin φ  (4)

FIG. 17 is a diagram showing an example of a captured image 2 h.

FIG. 17 shows an example of a fisheye-lens captured image 2 h that iscaptured in the case of installing the fisheye lens imaging apparatus1501 of the equidistant projection type, under a condition equivalent tothe condition for the imaging apparatus 101 that captures the capturedimage 2 in FIG. 3.

As shown in FIG. 17, the image captured using the fisheye lens of theequidistant projection type has a distortion peculiar to the fisheyelens, but it is possible to generate, by performing the conversion asbelow, an image having the distortion corrected.

FIG. 18 is a diagram representing a projection state of the incidentlight, in the lens of a central projection type.

Here, when considering the lens of the central projection type having afocal length f2, a distance L2 from a center of the imaging plane isrepresented by Expression (5) below, and a relationship between L2 andthe imaging plane coordinates P2=(x2, y2) is represented by Expressions(6) and (7).

[Math. 2]

L₂=f₂ tan θ  (5)

x₂=L₂ cos φ  (6)

y₂=L₂ sin φ  (7)

Here, according to Expressions (1), (2), and (5), a relationship betweenL1 and L2 is represented by a relational expression (8) as below.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 3} \right\rbrack & \; \\{L_{2} = {f_{2}{\tan \left( {\sin^{- 1}\left( \frac{L_{1}}{f_{1}} \right)} \right)}}} & (8)\end{matrix}$

According to Expressions (6), (7), and (8), when the focal length ofeach lens is known and the incident angle and the zenith angle of thelight are equal, it is possible to convert the imaging plane coordinatesP2 corresponding to the imaging plane coordinates P1. When the internalparameters of the camera are known, it is possible to convert theimaging plane coordinates to image coordinates, thus allowing the imagecaptured using the fisheye lens of the equidistant projection type intoan image captured using a lens of the central projection type.

FIG. 19 is a diagram showing an example of the corrected image.

FIG. 19 is an example of a distortion-corrected image 2 i generated byconverting the image in FIG. 17 using the conversion described above.FIG. 19 shows that an image captured using a wide-angle lens, which isthe same image as the captured image 2 in FIG. 3, can be generated bydistortion correction.

As described above, by including an image distortion correcting unit1502, it is possible to generate an image without distortion (capturedimage 2 i), from the fisheye-lens captured image 2 h that is capturedusing the fisheye lens imaging apparatus 1501.

Accordingly, subsequently, by performing the same processing as that ofthe first embodiment, it is possible to generate an image in which tiltsof the objects or people are unified regardless of their positions inthe captured image 2 i.

Note that the next operation may be performed as follows, for example.

In other words, the captured image 2 may be captured (see the imagingapparatus 101 in FIG. 1, FIGS. 2 and 3, and so on).

In addition, the captured image 2 may be an image obtained by capturinga lower direction from a higher point than the ground (FIG. 2 and so on)at the angle of depression 201 (FIGS. 8 and 2, and so on) with respectto the horizontal direction (direction x in FIG. 8).

In other words, for example, the imaging apparatus 101 is a monitoringcamera and so on.

However, it is possible to consider performing the processing asdescribed earlier, such as human detection, on the captured image 2 thuscaptured.

In other words, for example, the captured image 2 includes not only oneimage portion (for example, the first image portion 4 aI in FIG. 7 andso on) but also the other image portion (the other image portion 4 bIand so on).

The one image portion is an image portion captured by viewing thevertical direction (the direction from the foot to the head of theperson, that is, a negative direction in the z direction in FIG. 8) asthe top-bottom direction 2 y (the direction from the bottom to the top).

Then, the other image portion is captured by viewing the verticaldirection (the negative direction in the z direction in FIG. 8) asanother direction (diagonal direction) (a diagonally right upwarddirection 4 bd in the other image portion 4 bI in FIG. 7) that is adirection other than the direction of the top-bottom direction 2 y (thedirection from the bottom to the top) as described earlier.

This might complicate the configuration, as described earlier, forexample, as a result of requiring an extra identifier appropriate forthe processing on the other image portion.

Thus, the extended image 3 (see the lower part of FIG. 7 and so on) maybe generated from the captured image 2 (the image extension processingunit 103, ST1403 in FIG. 14).

In other words, as a result of this generation, the other direction asdescribed above that is the direction (direction 4 bd) in which an imagein the vertical direction is captured, may be changed from the direction(direction 4 bd) in the other image portion to a direction 5 bd (thedirection of the line 5 bL in the lower part of FIG. 10) in thegenerated extended image 3 that is the same direction as the top-bottomdirection 3 y of the extended image 3.

Note that the extended image 3 may be, for example, an image and thelike generated by adding, to the captured image 2, deformation caused bythis change.

Note that for example, a person who appears in the extended image 3 andhas the direction 5 bd as the direction of the person (from the foot tothe head) (see the person 100 h in FIG. 2) is the same person as theperson who appears in the captured image 2 and has the direction 4 bd asthe direction of the person.

Note that specifically, as described earlier, this generation may beperformed by sampling each pixel along the direction 4 bd (half line803L in FIG. 10).

Note that for example, the captured image 2 may be converted into theextended image 3.

Then, the processing such as human detection may be performed on theextended image 3 after change, without performing human detection andother processing on the captured image 2 before change.

With this, the processing is performed on the extended picture 3, so asto avoid, for example, requiring an extra identifier, thus simplifyingthe configuration.

However, it is noticed that the configuration might not be fullysimplified when, for example, complicated processing is performed in aprocess of, for example, analyzing the captured image 2 for identifyingthe direction of the direction 4 bd in which an image of the verticaldirection (direction z in FIG. 8) is captured.

In this context, it is noticed that the captured image 2 includes theunder-camera point 801 (FIGS. 8 and 9) at which an image of the verticaldirection (z direction in FIG. 8) of the imaging apparatus 101 iscaptured.

More specifically, it is noticed that a direction from this under-camerapoint 801 to the other image portion (the other image portion 4 bI) (thedirection 803L of the half line 803 in FIG. 10) is the same direction asthe direction 4 bd (FIG. 7 and so on) in the other image portion, inwhich an image of the vertical direction (direction z) is captured.

Thus, the under-camera point 801 may be specified (the correspondingpoint calculating unit 105, ST1401).

More specifically, with this, the direction from the specifiedunder-camera point 801 to the other image portion (the direction 803L ofthe half line 803) may be specified as the direction 4 bd in which animage of a vertically upward direction is captured.

Then, based on the specified direction 4 bd, the extended image 3 inwhich this direction 4 bd is changed (converted) to the direction 3 ydescribed earlier may be generated.

This simply requires only simple processing such as specifying theunder-camera point 801, so that no complicated processing is performed,thus sufficiently (significantly) reducing the configuration.

Note that for example, the following operation may be performed in acertain aspect.

In other words, the imaging apparatus 101 may capture the first image(captured image) 2.

The first image 2 to be captured may include the first image portion 4bI (FIG. 7).

In the first image portion 4 bI to be included, the person 4 b appears.

On the other hand, the imaging apparatus 101 that performs imagecapturing may be installed to be oriented downward with respect to thehorizontal direction (direction x in FIG. 2) at the angle of depression201. Note that for example, the direction of the optical axis of theimaging apparatus 101 is a direction downward at the angle 201 likethis.

Accordingly, in the image portion 4 bI described earlier, the direction4 bd from the foot to the head of the person 4 b captured in this imageportion 4 bI is different and displaced from the direction 4 ad from thefoot to the head of a person 5 a that appears in the other image portion4 aI in the first image 2.

Note that for example, the direction 4 ad described above, as shown inFIG. 7 and so on, is the same direction as the top-bottom direction 2 yof this first image 2, and the like.

Thus, the point (under-camera point) 801 as below in the captured firstimage 2 can be focused.

The point 801 is a point at which a line extended vertically downwardfrom the imaging apparatus 101 (see the unit under-camera vector 806 inFIG. 8) intersects with the plane (imaging plane) 807 which is a surfaceof the first image 2 and on which the first image 2 is projected(appears).

Specifically, the direction 803L of the line (straight line) 803 passingthrough this point 801 is the same direction as the direction 4 bd ofthe person 4 b in the one image portion 4 bI, which is the displaceddirection as described earlier.

Thus, the direction 803L of the line 803 described above may bespecified by the specifying unit 105 x (FIG. 1).

More specifically, for example, by specifying a plurality of pixelsincluded in the line 803 and specifying, as the line 803, a line made upof the specified pixels, the direction of the line 803 may be specified.

The second image 3 may be generated from the first image 2.

The generated second image 3 includes the second image portion 5 bI.

The included second image portion 5 bI is the image portion generatedfrom converting the specified direction (direction 803L) in the firstimage portion 4 bI described earlier to the next direction (direction 3y).

The direction is the direction 5 ad from the foot to the head of theperson 5 a in the other image portion 5 aI that is other than the imageportion 5 bI described above and includes an image of the person 5 athat is the same person as the person 5 a that appears in the otherimage portion 4 aI in the second image 3 as described earlier.

Note that this direction 5 ab is, for example, the same direction as thetop-bottom direction 3 y of the second image 3.

The position of the person 5 b in the second image 3 may be detected asthe position of the person 101 h (FIG. 2) by the human detector.

Note that this human detector is, for example, a human detector 100 d(FIG. 2) included in the image conversion processing unit 100.

This allows, for example, simplifying the configuration of the humandetector which detects the person 5 b, for such a reason that this humandetector is the same as the human detector detecting the person 5 a asdescribed above.

Note that for example, the imaging apparatus 101 may capture the firstimage 2 with the lens 101L included in the imaging apparatus 101.

The lens 101L described above may be a wide-angle lens.

That is, for example, in 35 mm equivalent, a horizontal angle of astandard lens having a focal length of 60 mm is approximately 39.6degrees.

The wide-angle lens as described earlier refers to a lens having a widerfield angle than the field angle of such a standard lens.

Note that data (the angle 201 described above and so on) indicating acorrespondence relationship between pixels in the first image 2 andpixels in the second image 3 may be held by the specifying unit 105 x(parameter holding unit 104).

The second image 3 to be generated may include, as next pixels, pixelshaving pixel values specified from pixel values of the pixels includedin the first image 2. The pixels are pixels associated with the pixelsin the first image 2 as described above, by the correspondencerelationship indicated by the held data.

Note that the pixel values specified from the pixel values of the pixelsincluded in the first image 2 may be the same pixel values as the pixelvalues of the first image 2 as described above, or may be an averagepixel value of the above-described pixel values of the pixels includedin the first image 2 and pixel values of pixels neighboring the pixels.

With this, the second image 3 may be generated by converting thedirection 4 bd of the person 100 h in the first image portion 4 bI inthe first image 2 to the direction 5 bd of the person 100 h in thesecond image portion 5 bI in the second image 3.

Note that the data 101 d indicating the angle 201 may be obtained by theobtaining unit 100 g in the image conversion processing unit 100.

This obtainment, for example, may be performed when the angle 201 atwhich the imaging apparatus 101 is installed is changed, to the angle201 b, from the one angle 201 a that is the angle other than the otherangle 201 b, or may be performed when installing the imaging apparatus101.

When the indicated angle 201 is the one angle 201 a, the processingdescribed earlier may be performed based on the intersection 801 in theimaging plane 807 for installing the imaging apparatus 101 to beoriented downward at the one angle 201 a.

When the angle 201 is the other angle 201 b, the processing describedearlier may be performed with reference to the intersection 801 in theimaging plane 807 for installing the imaging apparatus 101 downward atthe other angle 201 b.

With this, in both cases of installation at the one angle 201 a and theinstallation at the other angle 201 b, an appropriate second image 3 isgenerated, thus reliably performing appropriate generation of the secondimage 3.

Note that the obtained data 101 d may be, for example, the data inputinto the image conversion processing unit 100 by the user operating onthe image conversion processing unit 100.

On the other hand, this data 101 d may be obtained from the imagingapparatus 101.

For example, the imaging apparatus 101 may include a detecting unit 101s which detects an angle of depression of the imaging apparatus 101 suchas a gyroscope.

The obtaining unit 100 g may obtain the data 101 d which indicates, asthe angle 201 described above, the angle detected by this detecting unit101 s.

Note that, for example, when the data 101 d is obtained, the data to beheld may be updated with appropriate data when the angle at which theimaging apparatus 101 is placed is the angle 201 indicated by theobtained data 101 d.

After this updating, as the processing described earlier, the processingbased on the updated data may be performed.

With this, it is not necessary to obtain data 101 d again after theupdating above, thus allowing performing the processing relativelyeasily.

Note that each half line 803 is considered as follows.

Specifically, in the generated second image 3; each line 5 bL (FIG. 10)has the same order (as described earlier) as the order of acorresponding half line 803.

The pixels in the line 5 bL having the same order in the second image 3are pixels specified from the pixels included in the half line 803 inthe first image 2, such as the pixels in the half line 803.

Note that for example, the pixels of the line 5 bL can be specified fromthe pixels in the half line 803. The latter a pixel in the half line 803is located among the plurality of pixels in the half line 803 (in orderof being further away from the point 801), the latter the pixel islocated in the line 5 bL to be specified from the pixel (in order ofbeing on a more positive side in the direction 3 y).

For example, accordingly, the generated second image 3 may include animage portion of the person (person 5 a, 5 b), in terms of the person(person 4 a, 4 b) included in any image portion of the first image 2(image portion 4 aI, 4 bI).

This allows, in the second image 3, imaging any person and detecting anyperson successfully, thus allowing reliable performance of humandetection.

In addition, the following operation may be performed.

Specifically, as described above, a first pixel 9 b 1 (pixel 9 bn, 9 bf)in FIG. 12 corresponds to each of the pixels in the line 803 passingthrough the point 801 (FIG. 10 and so on) in the first image 2.

In addition, there is a second pixel 9 b 2 in the line 803.

For example, a pixel in the second image 3 which is specified from thesecond pixel 9 b 2 is adjacent to a pixel in the second image 3 which isspecified from the first pixel 9 b 1, from the top-bottom direction 3 yof the second image 3.

There is a first interval 1201 between the first pixel 9 b 1 and thesecond pixel 9 b 2.

As shown in FIG. 12, for example, the interval 1201 at the first pixel 9b 1 that is a pixel 9 bf located relatively far from the point 801 maybe wider than the interval 1201 at the first pixel 9 b 1 that is a pixel9 bn located relatively close.

This allows generating, as the second image 3, a second image 3 (see thesecond image 3 q in FIG. 12) that is more appropriate than a secondimage 3 (see the second image 3 p in FIG. 11) generated when theinterval 1201 is not wider, thus allowing generating the second image 3more appropriately.

On the other hand, there is also a third pixel 9 b 3 in another line 803that is other than the line 803 including the first pixel 9 b 1 (pixels9 bn, 9 bf) as described above.

For example, a pixel in the second image 3 which is specified from thisthird pixel 9 b 3 is adjacent to the pixel specified from the firstpixel 9 b 1 as described above, from the left-right direction 3 x of thesecond image 3.

An interval 1202 is provided between the first pixel 9 b 1 and thisthird pixel 9 b 3.

The interval 1202 at the pixel 9 b 1 (pixel 9 bf) located farther iswider than the interval 1202 at the first pixel 9 b 1 (pixel 9 bn)located closer (see FIG. 12).

A ratio between the intervals 1201 and 1202 in the first pixel 9 b 1that is farther (pixel 9 bf) may be the same as the ratio between theintervals 1201 and 1202 at the first pixel 9 b 1 that is closer (pixel 9bn).

Note that this ratio, for example, is a ratio of “1:1” as shown in twosymbol “Δr”s shown in FIG. 12.

In other words, the interval 1201 at each first pixel 9 b 1 (closerpixel 9 bn, farther pixel 9 bf) may be the same as the interval 1202 atthe first pixel 9 b 1.

With this, the interval 1201 at the first pixel 9 b 1 that is farther(pixel 9 bf) may be wider than the interval 1201 at the first pixel 9 b1 that is closer (pixel 9 bn).

Note that, for example, the specifying unit 105 x may specify a pixel inthe first image 2 which corresponds to the pixel in the second image 3(see the corresponding point calculating unit 105).

The specifying unit 105 x may specify the pixel value of the pixel inthe second image 3 from the pixel value of a pixel included in the firstimage 2 and corresponding to the pixel in the second image 3.

Note that two or more pixels may be specified as the pixels in the firstimage 2 which correspond to the pixel in the second image 3, and thepixel values of the pixels in the second image 3 may be specified fromthe pixel values of the two or more pixels.

Note that for example, the specifying unit 105 x may hold the dataindicating first coordinates on the imaging plane 807, a normal vectorof the imaging plane 807, second coordinates of the imaging apparatus101, and an optical axis direction 804L of the imaging apparatus 101(see the parameter holding unit 104).

The point 801 may be specified as a position of the intersection betweena plane of the indicated normal vector passing through a point of theindicated first coordinates and a line (straight line) passing throughthe indicated second coordinates and having the indicated direction804L.

Note that this specifying may be performed by calculating a formula fordetermination of this kind in a mathematical area.

Note that the point 801 thus specified may be located at one positionwhen the indicated direction 804L is one direction that is the directionwhen the angle of depression 201 is one angle 201 a, and may be locatedat the other position when the indicated direction 804L is the otherdirection in the case where the angle of depression 201 is the otherangle 201 b.

As the direction 803L, when one position is specified as the point 801,one direction 803La of one line 803 passing through the one position maybe specified, and another direction 803Lb of another line 803 passingthrough another position may be specified when the other position isspecified.

This allows avoiding the direction 9 xd passing through a centerposition 9 x of the first image 2 from being specified, and specifying adirection other than the direction 4 bd in which an image of the upwarddirection of the vertical direction z of the space 101 r in which theperson 100 h is present is captured, thus avoiding an inappropriatedirection from being specified.

In other words, the direction 803La (the one direction 803La or theother direction 803Lb) of the line 803 (the one line or the other line)passing through the point 801 below in the vertical direction of theimaging apparatus 101 is specified, so that the appropriate direction isspecified.

With this, assuming that a person 100 h is located relatively far fromthe imaging apparatus 101, such as a leftmost person 100 h among thethree people 100 h in FIG. 2, an appropriate operation is performeddespite the direction 804L of the optical axis being directed diagonallydownward with respect to the horizontal direction x at the angle ofdepression 201 instead of being directed downward in the verticaldirection z. In other words, the operation allows preventing theinappropriate direction 9 xd from being specified, and specifying theappropriate direction 4 bd (the one direction 803La or the otherdirection 803Lb).

This leads to generating the second image 3 including an appropriatesecond image portion 5 bI based on the specified direction, and thesecond image portion 5 bI is detected from this second image 3. Thisprevents complexity of the configuration of the detector (human detector100 d) which detects, from the image (second image 3), an image portion(the second image portion 5 bI) including an image of the person 100 h.

This prevents a complicated configuration despite the optical axisdirection 804L being diagonally downward, and allows simplifying theconfiguration.

Moreover, as described above, the following operation is performedalthough not only the one angle 201 a but also the other angle 201 b canbe the angle 201 when the direction 804L of the optical axis is directeddiagonally downward.

The operation not only specifies the appropriate direction by specifyingthe one direction 803La as the direction 803L when the angle 201 is theone angle 201 a.

In other words, the appropriate direction is also specified when theangle 201 is the other angle 201 b, by specifying the other direction803Lb as the direction 803L.

This allows specifying the appropriate direction relatively reliably.

Thus, a plurality of configurations (the image extension processing unit103 and so on) are combined, and such combination produces a synergeticeffect. In contrast, the known prior art is missing part or all of theseconfigurations, so that no synergetic effect is produced. In thisrespect, the technique according to the present invention is differentfrom the prior art.

Note that the present invention can be realized not only as an apparatusbut also as: a method including, as steps, processing units included inthe apparatus; a program for causing a computer to execute these steps;a computer-readable recording medium such as a compact disc read onlymemory (CD-ROM) on which the program is recorded; information indicatingthe program; data or a signal. Moreover, such program, information,data, and signal may be distributed via a communication network such asthe Internet.

Although only some exemplary embodiments of the present invention havebeen described in detail above, those skilled in the art will readilyappreciate that many modifications are possible in the exemplaryembodiments without materially departing from the novel teachings andadvantages of the present invention. Accordingly, all such modificationsare intended to be included within the scope of the present invention.

INDUSTRIAL APPLICABILITY

In an output image (extended image 3 in FIG. 10), tilts of objectsstanding upright with respect to a ground are unified, so that objectdetection, particularly human detection, requires only one featurequantity.

It is possible to improve convenience of a system 100 s including animaging apparatus 101 and an image conversion processing unit 100.

It is possible to prevent a complicated configuration despite adirection 804L of an optical axis being directed diagonally downward, tosimplify the configuration.

1. An image generating apparatus which generates a second image that is an image converted from a first image captured by an imaging apparatus placed to be oriented downward at an angle of depression with respect to a horizontal direction, said image generating apparatus comprising: a specifying unit configured to specify a straight line included in the first image and passing through an intersection between an imaging plane and a vertical vector having a start point at the imaging apparatus; and an image extension processing unit configured to generate the second image by sampling pixels of the first image along the specified straight line.
 2. The image generating apparatus according to claim 1, wherein said specifying unit is configured to specify the pixels included in the specified straight line, and the generated second image includes the sampled and specified pixels as pixels included in a straight line in a top-bottom direction of the second image.
 3. The image generating apparatus according to claim 2, wherein the specified straight line is a straight line passing through a first intersection that is the intersection and a second intersection, and the second intersection is an intersection between the imaging plane and a third vector generated by rotating a second vector around a first vector as a rotational axis, the first vector being the vertical vector, and the second vector having a start point at the imaging apparatus.
 4. The image generating apparatus according to claim 3, wherein the first image includes a pixel that is located at the second intersection with the third vector generated by rotating the second vector at each of different angles, and the second image includes, as pixels in a straight line in a left-right direction of the second image, a plurality of pixels including the pixel that is located at the second intersection at each of the different angles and is included in the first image.
 5. The image generating apparatus according to claim 4, wherein said specifying unit is configured to generate the third vector at each of the different angles and calculate the second intersection in the generated third vector.
 6. The image generating apparatus according to claim 5, comprising an obtaining unit configured to obtain data indicating the angle of depression, wherein the second intersection calculated at each of the different angles is an intersection with the imaging plane at the angle of depression indicated by the obtained data.
 7. The image generating apparatus according to claim 6, wherein the angle of depression indicated by the obtained data is an angle detected by a gyroscope included in the imaging apparatus.
 8. The image generating apparatus according to claim 2, comprising a detecting unit configured to detect a position of an image portion of a person as a location of the person, the image portion being included in the straight line in the top-bottom direction of the second image.
 9. The image generating apparatus according to claim 1, wherein the first image is captured by the imaging apparatus using a wide-angle lens included in the imaging apparatus.
 10. The image generating apparatus according to claim 1, wherein the first image is captured by the imaging apparatus using a fisheye lens included in the imaging apparatus, said image generating apparatus comprises an image distortion correcting unit configured to correct the first image before correction which includes a distortion derived from the fisheye lens to the first image after the correction which does not include the distortion, and the second image is not generated from the first image before the correction, but is generated from the first image after the correction.
 11. The image generating apparatus according to claim 1, wherein the specified straight line includes a first pixel and a second pixel, the generated second image includes one pixel which is specified from the first pixel and another pixel which is adjacent to the one pixel and is specified from the second pixel, and the more distant the first pixel is from the intersection of the vertical vector, the wider an interval between the first pixel and the second pixel is.
 12. The image generating apparatus according to claim 11, wherein the second image includes a pixel specified from a third pixel which is adjacent to the one pixel from a right-left direction of the second image and is included in the first image, and the interval is a first interval that is different from a second interval which is an interval between the first pixel and the third pixel, and a ratio of the first interval with respect to the second interval at the first pixel located relatively distant from the intersection of the vertical vector is equal to a ratio of the first interval with respect to the second interval at the first pixel located relatively close to the intersection.
 13. The image generating apparatus according to claim 1, wherein the captured first image includes an image portion including an image of a person, a direction from a foot to a head of the person in the captured first image is a different direction from a top-bottom direction of the first image, the different direction coincides with a direction of the straight line passing through the intersection between the imaging plane and the vertical vector and through a position of the image portion, said specifying unit is configured to specify the straight line passing through the intersection, so as to specify a direction of the specified straight line as the direction of the person, the generated second image includes an image portion including an image of the same person as the person in the captured first image, and the included image portion is an image portion generated by converting the specified direction in the image portion included in the first image to a top-bottom direction of the second image.
 14. An image generating method for generating a second image that is an image converted from a first image captured by an imaging apparatus placed to be oriented downward at an angle of depression with respect to a horizontal direction, said image generating method comprising: specifying a straight line included in the first image and passing through an intersection between an imaging plane and a vertical vector having a start point at the imaging apparatus; and generating the second image by sampling pixels of the first image along the specified straight line.
 15. A non-transitory computer-readable recording medium which holds a program for generating a second image that is an image converted from a first image captured by an imaging apparatus placed to be oriented downward at an angle of depression with respect to a horizontal direction, the program causing a computer to execute: specifying a straight line included in the first image and passing through an intersection between an imaging plane and a vertical vector having a start point at the imaging apparatus; and generating the second image by sampling pixels of the first image along the specified straight line. 